Fair and Balanced? Quantifying Media Bias through Crowdsourced Content Analysis Ceren Budak*

It is widely thought that news organizations exhibit ideological bias, but rigorously quantifying such slant has proven methodologically challenging. Through a combination of machinelearning and crowdsourcing techniques, we investigate the selection and framing of political issues in fifteen major US news outlets. Starting with 803,146 news stories published over twelve months, we first used supervised learning algorithms to identify the 14 percent of articles pertaining to political events. We then recruited 749 online human judges to classify a random subset of 10,502 of these political articles according to topic and ideological position. Our analysis yields an ideological ordering of outlets consistent with prior work. However, news outlets are considerably more similar than generally believed. Specifically, with the exception of political scandals, major news organizations present topics in a largely nonpartisan manner, casting neither Democrats nor Republicans in a particularly favorable or unfavorable light. Moreover, again with the exception of political scandals, little evidence exists of systematic differences in story selection, with all major news outlets covering a wide variety of topics with frequency largely unrelated to the outlet’s ideological position. Finally, news organizations express their ideological bias not by directly advocating for a preferred political party, but rather by disproportionately criticizing one side, a convention that further moderates overall differences. Ceren Budak is an assistant professor in the School of Information at the University of Michigan, Ann Arbor, MI, USA. Sharad Goel is an assistant professor in the Management Science and Engineering Department at Stanford University, Palo Alto, CA, USA. Justin M. Rao is a senior researcher at Microsoft Research, Redmond, WA, USA. The authors thank Seth Flaxman, Matthew Salganik, and Sid Suri for comments. *Address correspondence to Ceren Budak, University of Michigan, School of Information, 105 S. State St. Ann Arbor, MI 48109-1285, USA; e-mail: [email protected]. Public Opinion Quarterly, Vol. 80, Special Issue, 2016, pp. 250–271 doi:10.1093/poq/nfw007